Document uv-with-Jupyter workflows #7625

charliermarsh · 2024-09-22T17:56:29Z

Summary

This is a work-in-progress as I actually want to change some behaviors here.

Closes #6329.

charliermarsh · 2024-09-22T18:03:30Z

docs/guides/integration/jupyter.md

+
+    Since the Jupyter server is running in an isolated virtual environment, `!uv pip install` will install package's
+    into _Jupyter's_ environment, not the project environment. Such dependencies may disappear on subsequent `jupyter`
+    invocations. To install packages into the project environment, use `!uv add`.


Actually, it looks like this is solvable using kernel ipython kernel install (#6329 (comment)).

charliermarsh · 2024-09-22T18:15:38Z

docs/guides/integration/jupyter.md

+
+    Since the Jupyter server is running in an isolated virtual environment, `!uv pip install` will install package's
+    into _Jupyter's_ environment, not the project environment. Such dependencies may disappear on subsequent `jupyter`
+    invocations. To install packages into the project environment, use `!uv add`.


I sort of think we should not be discovering the tool environment here, though. Modifying the project environment isn't great, but modifying the tool environment is even worse! It means subsequent users of the tool have these same modifications. I'd propose that we skip tool and ephemeral environments when considering active virtual environments. \cc @zanieb

charliermarsh · 2024-09-22T19:17:50Z

docs/guides/integration/jupyter.md

+- Select the virtual environment that uv created. It will be named .venv/bin/python in this dropdown
+  (or maybe .venv\Scripts\python on windows)
+
+If you don't `uv add ipykernel`, the notebook will fail to execute with an error.


It would be nice if we could find a way to make this work without requiring that you install ipykernel into the project environment itself. (We don't want users to have to add ipykernel as a project dependency just to interact with their project in a notebook.)

Why is that not wanted?

But the 'Uv Run' kernelspec I posted in the other issue seems to be the most normal way to achieve exactly that - it runs the kernel using uv run --with ipykernel. Maybe a vscode extension is a better user interface, most of my questions are around the interface and if it's worth it (Because vscode already picks up .venv natively, is it worth the fuss with kernelspecs or other ways to do it - or can those external configuration ways of working be put on the long term plan, that is working on better uv + vscode integration to make these cases easy or automatic in the long run.)

Well, if I'm working on a project, and I want to play with that project in a notebook, I don't want to add notebook dependencies as dependencies to the project. I don't want them recorded in the lock, etc.

bluss · 2024-09-22T22:40:06Z

docs/guides/integration/jupyter.md

+with access to the project's virtual environment via the following:
+
+```console
+$ uv run --with jupyter jupyter notebook


I'm sure there are various opinions are about this, but jupyter notebook is the legacy interface and jupyterlab is the current interface. I don't know what relation other people have to this, would it give a better impression to use jupyterlab here? Command jupyter lab.

Yeah I went back and forth. I'm fine to use jupyter lab, no big deal.

I think that's the best, it's what jupyter promotes as #1

radames · 2024-09-23T18:26:28Z

Just one note, following a fresh new setup with vscode

uv init my-notebook
cd my-notebook
uv add --dev ipykernel
uv run ipython kernel install --user --name=uv
code .

!uv pip install diffusers
/usr/bin/sh: 1: uv: not found

The notebook kernel couldn't find the correct env path for uv nor pip , not sure if it's specific to my env however uv is set correct via my .zshrc and .profile . "$HOME/.cargo/env". Doing the same steps with pip doesn't lead to an issue

charliermarsh · 2024-09-23T18:28:41Z

Good call, I experienced the same (though I don't see that when running Jupyter directly).

radames · 2024-09-23T18:45:11Z

indeed uv run --with jupyter jupyter lab works, however it looks like there are still some PATH issues. pyproject.toml is not an error, just started it from a fresh dir

docs/guides/integration/jupyter.md

+    with `uv pip install ipykernel`.
+
+If you need to manipulate the project's environment from within the notebook, you may need to add
+`uv` as an explicit device dependency:


charliermarsh · 2024-09-23T18:46:23Z

Do you mean that !pip list and !nvidia-smi are missing?

charliermarsh · 2024-09-23T18:55:55Z

Does !nvidia-smi work in other Jupyter notebooks for you?

radames · 2024-09-23T19:12:05Z

Does !nvidia-smi work in other Jupyter notebooks for you?

yes, starting a jupyter lab from a new conda env, the env path are working correctly

conda create -n temp python=3.10 
pip install jupyterlab
jupyter lab

charliermarsh · 2024-09-23T23:03:45Z

I haven't been able to reproduce that unfortunately.

docs/guides/integration/jupyter.md

zanieb · 2024-09-25T23:22:45Z

docs/guides/integration/jupyter.md

+with access to the project's virtual environment via the following:
+
+```console
+$ uv run --with jupyter jupyter lab


Future: I wonder if we want uvx --with-project jupyter lab?

Yeah I kinda like that. The repeated jupyter would be nice to avoid.

docs/guides/integration/jupyter.md

zanieb · 2024-09-25T23:27:07Z

docs/guides/integration/jupyter.md

+Then, you can create the kernel for `project` with:
+
+```console
+$ uv run ipython kernel install --user --name=project


Could be cool to have a command that emits the project name, so you could do --name=$(uv ...) as a one-liner that actually works in any project.

docs/guides/integration/jupyter.md

zanieb

Small comments

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.15` -> `0.4.18` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.4.18`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0418) [Compare Source](astral-sh/uv@0.4.17...0.4.18) ##### Enhancements - Allow multiple source entries for each package in `tool.uv.sources` ([#7745](astral-sh/uv#7745)) - Add `.gitignore` file to `uv build` output directory ([#7835](astral-sh/uv#7835)) - Disable jemalloc on FreeBSD ([#7780](astral-sh/uv#7780)) - Respect `PAGER` env var when paging in `uv help` command ([#5511](astral-sh/uv#5511)) - Support `uv run -m foo` to run a module ([#7754](astral-sh/uv#7754)) - Use a top-level output directory for `uv build` in workspaces ([#7813](astral-sh/uv#7813)) - Update `uv init --package` command to match project name ([#7670](astral-sh/uv#7670)) - Add a custom suggestion for `uv add dotenv` ([#7799](astral-sh/uv#7799)) - Add detailed errors for `tool.uv.sources` deserialization failures ([#7823](astral-sh/uv#7823)) - Improve error message copy for failed builds ([#7849](astral-sh/uv#7849)) - Use `serde-untagged` to improve some untagged enum error messages ([#7822](astral-sh/uv#7822)) - Use build failure hints for `dotenv` errors, rather than in `uv add` ([#7825](astral-sh/uv#7825)) ##### Configuration - Add `UV_NO_SYNC` environment variable ([#7752](astral-sh/uv#7752)) ##### Bug fixes - Accept `git+` prefix in `tool.uv.sources` ([#7847](astral-sh/uv#7847)) - Allow spaces in path requirements ([#7767](astral-sh/uv#7767)) - Avoid reusing cached downloaded binaries with `--no-binary` ([#7772](astral-sh/uv#7772)) - Correctly trims values during wheel WHEEL file parsing ([#7770](astral-sh/uv#7770)) - Fix `uv tree --invert` for platform dependencies ([#7808](astral-sh/uv#7808)) - Fix encoding mismatch between python child process and uv ([#7757](astral-sh/uv#7757)) - Reject self-dependencies in `uv add` ([#7766](astral-sh/uv#7766)) - Respect `tool.uv.environments` for legacy virtual workspace roots ([#7824](astral-sh/uv#7824)) - Retain empty extras on workspace members ([#7762](astral-sh/uv#7762)) - Use file stem when parsing cached wheel names ([#7773](astral-sh/uv#7773)) ##### Rust API - Make `FlatDistributions` public ([#7833](astral-sh/uv#7833)) ##### Documentation - Fix table of contents sizing ([#7751](astral-sh/uv#7751)) - GitLab Integration documentation ([#6857](astral-sh/uv#6857)) - Update documentation to setup-uv@v3 ([#7807](astral-sh/uv#7807)) - Use `uv publish` instead of twine in docs ([#7837](astral-sh/uv#7837)) - Fix typo in `projects.md` ([#7784](astral-sh/uv#7784)) ### [`v0.4.17`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0417) [Compare Source](astral-sh/uv@0.4.16...0.4.17) ##### Enhancements - Add `uv build --all` to build all packages in a workspace ([#7724](astral-sh/uv#7724)) - Add support for `uv init --script` ([#7565](astral-sh/uv#7565)) - Add support for upgrading build environment for installed tools (`uv tool upgrade --python`) ([#7605](astral-sh/uv#7605)) - Initialize a Git repository in `uv init` ([#5476](astral-sh/uv#5476)) - Respect `--quiet` flag in `uv build` ([#7674](astral-sh/uv#7674)) - Add context message before listing available tools in `uvx` ([#7641](astral-sh/uv#7641)) ##### Bug fixes - Don't create Python bytecode files during interpreter discovery ([#7707](astral-sh/uv#7707)) - Escape glob patterns in workspace member discovery ([#7709](astral-sh/uv#7709)) - Avoid prefetching source distributions with unbounded lower-bound ranges ([#7683](astral-sh/uv#7683)) ##### Documentation - Add `uv build` and `uv publish` to features overview ([#7716](astral-sh/uv#7716)) - Add documentation on cache versioning ([#7693](astral-sh/uv#7693)) - Spell out the names of the Docker images for easier copy-paste ([#7706](astral-sh/uv#7706)) - Document uv-with-Jupyter workflows ([#7625](astral-sh/uv#7625)) - Note that `uv lock --upgrade-package` retains locked versions ([#7694](astral-sh/uv#7694)) ### [`v0.4.16`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0416) [Compare Source](astral-sh/uv@0.4.15...0.4.16) ##### Enhancements - Add `uv publish` ([#7475](astral-sh/uv#7475)) - Add a `--project` argument to run a command from a project directory ([#7603](astral-sh/uv#7603)) - Display Python implementation when creating environments ([#7652](astral-sh/uv#7652)) - Implement trusted publishing for `uv publish` ([#7548](astral-sh/uv#7548)) - Respect lockfile preferences for `--with` requirements ([#7627](astral-sh/uv#7627)) - Unhide the `--directory` option ([#7653](astral-sh/uv#7653)) - Allow requesting free-threaded Python interpreters ([#7431](astral-sh/uv#7431)) - Show a dedicated PubGrub hint for `--unsafe-best-match` ([#7645](astral-sh/uv#7645)) - Add resolver error checking for conflicting distributions ([#7595](astral-sh/uv#7595)) ##### Bug fixes - Avoid adding double-newlines for CRLF ([#7640](astral-sh/uv#7640)) - Avoid retaining forks when `requires-python` range changes ([#7624](astral-sh/uv#7624)) - Determine if pre-release Python downloads should be allowed using the version specifiers ([#7638](astral-sh/uv#7638)) - Fix `link-mode=clone` for directories on Linux ([#7620](astral-sh/uv#7620)) - Improve Python executable name discovery when using alternative implementations ([#7649](astral-sh/uv#7649)) - Require opt-in to use alternative Python implementations ([#7650](astral-sh/uv#7650)) - Use the first pre-release discovered when only pre-release Python versions are available ([#7666](astral-sh/uv#7666)) ##### Documentation - Document environment variable that disables printing of virtual environment name in prompt ([#7648](astral-sh/uv#7648)) - Remove double whitespaces from the code ([#7623](astral-sh/uv#7623)) - Use anchorlinks rather than permalinks ([#7626](astral-sh/uv#7626)) ##### Preview features - Add build backend scaffolding ([#7662](astral-sh/uv#7662)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

charliermarsh added the documentation Improvements or additions to documentation label Sep 22, 2024

charliermarsh force-pushed the charlie/jupyter branch from c78217a to 38f0dc1 Compare September 22, 2024 18:01

charliermarsh commented Sep 22, 2024

View reviewed changes

charliermarsh mentioned this pull request Sep 22, 2024

Consider adding documentation on using uv ... to use a .venv in Jupyter notebooks #6329

Closed

charliermarsh force-pushed the charlie/jupyter branch from 38f0dc1 to fa6852e Compare September 22, 2024 18:32

charliermarsh commented Sep 22, 2024

View reviewed changes

bluss reviewed Sep 22, 2024

View reviewed changes

charliermarsh force-pushed the charlie/jupyter branch from fa6852e to 2f86317 Compare September 23, 2024 18:26

charliermarsh force-pushed the charlie/jupyter branch from 2f86317 to bde82ad Compare September 23, 2024 18:31

charliermarsh requested a review from zanieb September 23, 2024 18:34

charliermarsh force-pushed the charlie/jupyter branch from bde82ad to da19f01 Compare September 23, 2024 18:34

bluss reviewed Sep 23, 2024

View reviewed changes

docs/guides/integration/jupyter.md Outdated

with `uv pip install ipykernel`.

If you need to manipulate the project's environment from within the notebook, you may need to add

`uv` as an explicit device dependency:

This comment was marked as resolved.

Sign in to view

charliermarsh force-pushed the charlie/jupyter branch from da19f01 to 1f85ef5 Compare September 23, 2024 18:56

zanieb reviewed Sep 25, 2024

View reviewed changes

docs/guides/integration/jupyter.md Outdated Show resolved Hide resolved

zanieb reviewed Sep 25, 2024

View reviewed changes

docs/guides/integration/jupyter.md Outdated Show resolved Hide resolved

zanieb reviewed Sep 25, 2024

View reviewed changes

docs/guides/integration/jupyter.md Outdated Show resolved Hide resolved

zanieb reviewed Sep 25, 2024

View reviewed changes

docs/guides/integration/jupyter.md Outdated Show resolved Hide resolved

zanieb reviewed Sep 25, 2024

View reviewed changes

docs/guides/integration/jupyter.md Outdated Show resolved Hide resolved

zanieb reviewed Sep 25, 2024

View reviewed changes

docs/guides/integration/jupyter.md Outdated Show resolved Hide resolved

zanieb reviewed Sep 25, 2024

View reviewed changes

docs/guides/integration/jupyter.md Show resolved Hide resolved

zanieb approved these changes Sep 25, 2024

View reviewed changes

charliermarsh added 2 commits September 25, 2024 20:33

Document uv-with-Jupyter workflows

38db73a

Second round

3352698

charliermarsh force-pushed the charlie/jupyter branch from 1f85ef5 to 3352698 Compare September 26, 2024 00:54

charliermarsh enabled auto-merge (squash) September 26, 2024 00:54

charliermarsh merged commit 4ba0e56 into main Sep 26, 2024
50 checks passed

charliermarsh deleted the charlie/jupyter branch September 26, 2024 00:55

BrewTestBot mentioned this pull request Sep 27, 2024

uv 0.4.17 Homebrew/homebrew-core#192137

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document uv-with-Jupyter workflows #7625

Document uv-with-Jupyter workflows #7625

charliermarsh commented Sep 22, 2024 •

edited

Loading

charliermarsh Sep 22, 2024

charliermarsh Sep 22, 2024

charliermarsh Sep 22, 2024

bluss Sep 23, 2024 •

edited

Loading

charliermarsh Sep 23, 2024

bluss Sep 22, 2024

charliermarsh Sep 22, 2024

bluss Sep 25, 2024 •

edited

Loading

radames commented Sep 23, 2024

charliermarsh commented Sep 23, 2024

radames commented Sep 23, 2024

This comment was marked as resolved.

charliermarsh commented Sep 23, 2024

charliermarsh commented Sep 23, 2024

radames commented Sep 23, 2024

charliermarsh commented Sep 23, 2024

zanieb Sep 25, 2024

charliermarsh Sep 26, 2024

zanieb Sep 25, 2024

zanieb left a comment

Document uv-with-Jupyter workflows #7625

Document uv-with-Jupyter workflows #7625

Conversation

charliermarsh commented Sep 22, 2024 • edited Loading

Summary

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bluss Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bluss Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

radames commented Sep 23, 2024

charliermarsh commented Sep 23, 2024

radames commented Sep 23, 2024

This comment was marked as resolved.

charliermarsh commented Sep 23, 2024

charliermarsh commented Sep 23, 2024

radames commented Sep 23, 2024

charliermarsh commented Sep 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb left a comment

Choose a reason for hiding this comment

charliermarsh commented Sep 22, 2024 •

edited

Loading

bluss Sep 23, 2024 •

edited

Loading

bluss Sep 25, 2024 •

edited

Loading